Papers by Tatsunori B. Hashimoto
Unifying Human and Statistical Evaluation for Natural Language Generation (N19-1)
Copied to clipboard
| Challenge: | Human evaluation captures quality but fails to capture diversity . statistical evaluation fails to catch models that plagiarize from training set . |
| Approach: | They propose a framework which evaluates both diversity and quality based on the optimal error rate of predicting whether a sentence is human-generated. |
| Outcome: | The proposed framework evaluates diversity and quality on summarization and chit-chat dialogue. |
Improved Natural Language Generation via Loss Truncation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing training procedures for neural language models seek to match the underlying distribution, leading to models that replicate unwanted behaviors such as hallucination during generation. |
| Approach: | They propose a procedure which optimizes for distinguishability by removing high log loss examples from a corpus to minimize the log loss. |
| Outcome: | Empirically, the proposed procedure outperforms baselines on distinguishability on a summarization task and shows that the models outperformed human references on the summarizing task. |
DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021.naacl-main)
Copied to clipboard
| Challenge: | Meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution . meta-learners tend to overfit their adaptation mechanism and datasets are heterogeneous . |
| Approach: | They propose a method for decomposing datasets into Reasoning Categories to form additional high quality tasks. |
| Outcome: | The proposed method improves the accuracy of meta-learners by 1.5-4% across four few-shot NLI problems. |
Distributionally Robust Language Modeling (D19-1)
Copied to clipboard
| Challenge: | Language models are generally trained on data spanning a wide range of topics but might be applied to an unknown target distribution. |
| Approach: | They propose a distributionally robust optimization procedure which minimizes the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution. |
| Outcome: | The proposed method reduces the loss of the model over the worst-case mixture of topics with sufficient overlap with the training distribution. |
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies suggest that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. |
| Approach: | They construct cloze-like masks using task-specific lexicons to explain their results . they show that the majority of performance gains come from generic masks that are not associated with the lexical . |
| Outcome: | The proposed method outperforms a classic method for unsupervised parsing. |